United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, l'atint and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/540,315 06/21/2005 David AEves GB 020255 1627 



21127 7590 03/03/2009 

RISSMAN JOBSE HENDRICKS & OLIVERIO, LLP 
100 Cambridge Street 
Suite 2101 

BOSTON, MA 021 14 



SHAH, PARAS D 



PAPER NUMBER 



NOTIFICATION DATE | DELIVERY MODE 
03/03/2009 ELECTRONIC 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 

Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the 
following e-mail address(es): 

m^en-aimeC" 1 rhoiplaw.com 
cjosephC" 1 rhoiplaw.com 
info @ rhoiplaw.com 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/540,315 


Applicant(s) 

EVES ET AL. 


Examiner 

PARAS SHAH 


Art Unit 

2626 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 12/15/2008 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-22 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

Claim(s) is/are allowed. 

6) |EI Claim(s) 1-22 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) ^| Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 

Paper No(s)/Mail Date . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 200901 15 



Application/Control Number: 10/540,315 Page 2 

Art Unit: 2626 

DETAILED ACTION 

1 . In view of the Appeal Brief filed on 1 2/1 5/2008, PROSECUTION IS HEREBY 
REOPENED. A New Grounds for rejection is set forth below. 

To avoid abandonment of the application, appellant must exercise one of the 
following two options: 

(1 ) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply 
under 37 CFR 1.113 (if this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41 .31 followed 
by an appeal brief under 37 CFR 41 .37. The previously paid notice of appeal fee and 
appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth 
in 37 CFR 41 .20 have been increased since they were previously paid, then appellant 
must pay the difference between the increased fees and the amount previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by 
signing below: 



2. This communication is in response to the Appeal Brief filed on 12/15/2008. 
Claims 1-22 are pending and have been examined. The Applicants' amendment and 
remarks have been carefully considered, but they do not place the claims in condition 
for allowance. 
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3. All previous objections and rejections directed to the Applicant's disclosure and 
claims not discussed in this Office Action have been withdrawn by the Examiner. 

Response to Arguments 

4. Applicant's arguments (pages 9-1 3) filed on 1 2/1 5/2008 with regard to claims 1 - 
22 have been fully considered and they are persuasive. However, upon further 
consideration a new grounds for rejection has been made upon further search. 

Claim Rejections - 35 USC §112 

5. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

6. Claims 11,17, and 21 are rejected under 35 U.S.C. 112, second paragraph, as 
being indefinite for failing to particularly point out and distinctly claim the subject matter 
which applicant regards as the invention. Specifically, the limitation of "wherein the 
media fragments are unrelated to the audio signal prior to the obtaining act" is unclear 
as it contradicts the previous limitations. It is unclear as to how the media fragments are 
unrelated to the audio signal if the media fragments being obtained are related to the 
audio signal by way of the dramatic parameters or features. Hence, for purposes of 
compact prosecution the limitation was interpreted to mean the media fragment 
obtained is different from the audio input. 
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Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

8. Claims 1-3, 5, 7-10, 16-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Finn (WO 01/1 1495) in view of Mitton (US 6,355,869) in view of 
Nishimura et al. ("Music Signal Spotting Retrieval by a Humming Query Using Start 
Frame Feature Dependent Continuous Dynamic Programming"), hereinafter, Nishimura. 

As to claims 1, 17, 21, and 22: Finn discloses augmenting an audio signal (see 
Figure 1) comprising: 

receiving an audio signal (Figure 2, input search criteria steps 20 and 21) 

extracting features from said audio signal (see Figure 2 step 22, identify 
pitch of successive notes), 

generating a time based table of dramatic parameters according to the 
extracted features (see page 6, 3rd and 4th paragraph, and see Figure 3A, from 
the cited portion and the figure it can be seen that a time ordered sequence of 
pitches are extracted), and 

obtaining media fragments at least in part in dependence on the table of 
dramatic parameters (see page 15, line 10-33, comparison between query and 
database based on similarity) files and wherein the media fragments are 
unrelated to the audio signal prior to the obtaining act (see page 3, lines 5-7, only 
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a search criteria in input and a target file is retrieved, which is not the same as 
the audio input above), and 

outputting said media fragments (see page 21, lines 21-28, music file 
output or list displayed to user) 

However, Finn does not specifically teach a time-ordered table. 

Mitton does teach a time- ordered table (see Col. 5, lines 12-22, where 
Mitton discusses a pseudo wave file with a series of pitch coefficients for each 
frame and Figure 33). 

It would have been obvious to one skilled in the art at the time the 
invention was made to modify the audio search as taught by Finn, and use a 
time-ordered table as taught by Mitton, thus allowing a user to produce a musical 
score from a recording, as discussed by Mitton (see Col. 1 , lines 55-60). 

However, Finn in view of Mitton do not specifically teach wherein the 
media fragments are not audio media fragments. 

Nishimura does teach wherein the media fragments are not audio media 
fragments (see page 7, right column, sect. 5, first paragraph and see page 1, 
right column, lines 8-1 1 , where video is retrieved when video clips contain music 
and see Figure 7, lower right hand corner displays a related video based on 
query). 

It would have been obvious to one skilled in the art at the time the 
invention was made to modify the audio search as taught by Finn in view of 
Mitton, and retrieve video as taught by Nishimura, thus allowing a user to find 
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similar audio segments based on input signal attributes (see Nishimura 
Abstract)). 

As to claim 2, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Finn teaches features extracted from said audio signal 
include tempo (see page 25, lines 2, 15, key and tempo determined from the 
input and is used in search criteria (see page 23 lines 22-26, used in first pass 
matching). 

As to claim 3, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Mitton does teach generation of a time- ordered table (see 
Col. 5, lines 12-22, where Mitton discusses a pseudo wave file with a series of 
pitch coefficients for each frame and Figure 33). 

Furthermore, Finn discloses the table of dramatic parameters comprises 
retrieving a list of dramatic parameters and associated audio features (see page 
1 1 , lines 8-10, features from data are compared with search criteria, where the 
matching criteria of dramatic parameters is shown in page 25, lines 2 and 15), 
comparing and matching the extracted features with the retrieved associated 
audio features (see page 1 1 , lines 8-10, features from data are compared with 
search criteria), and inserting an entry comprising the dramatic parameter 
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associated with the audio feature (see page 23, lines 21 -page 25, lines 16, 
various criteria are determined in order to determined match, where the 
determination of the dramatic parameter is the inserting for matching purposes.) 

As to claim 5, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Finn teaches obtaining said media fragments comprises 
selecting a fragment from a store (see page 11, lines 8-1 0,m music files in 
database 9 and 10 used), said fragment being stored with an associated 
dramatic parameter which matches the respective entry in the table of dramatic 
parameters (see page 25, lines 2, 15, key and tempo determined from the input 
and is used in search criteria (see page 23 lines 22-26, used in first pass 
matching) . 

As to claim 7, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 5, above. 

Furthermore, Finn teaches receiving user input, said user input affecting 
said obtaining (see page 6, lines 13-18, user inputs a voice or a tune and see 
page 3, lines 1-8, based on user input a matching music is obtained). 
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As to claim 8, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Nishimura teaches the media fragments being video data, 
(see Figure 7, lower right hand corner, and see sect. 5, on page 7, where video is 
retrieved) 

As to claim 9, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Finn teaches Furthermore, Finn teaches further comprising 
storage for storing said media fragments (see page 1 1 , line 9, database 9 or 10). 

Furthermore, it would have been obvious to one of ordinary skilled in the 
art to have stored the audio signal at least temporarily as well in order to perform 
the extraction of features from the audio signal for comparison (see Finn, page 
6, lines 15-19). 

As to claim 10, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Finn teaches wherein said outputting comprises rendering 
said media fragments and said audio signal (see page 21 , lines 26-29, link to the 
media fragment is displayed which the user can select to hear. It is obvious that 
the computer system consists of a built in speaker to hear such results 
corresponding to the tune of the search query. Hence, the rendering of the audio 
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signal occurs by the rendering of a match that is found similar to the tune that 
was input.) 



As to claim 16, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

Furthermore, Finn teaches wherein combinations of extracted features 
have associated dramatic parameters (see page 25, line 15 and 16, where the 
tempo is based on mean note durations in seconds, i.e., the mean of the pitches 
for a specific duration determines the dramatic parameter). 



As to claims 18 and 22, Finn in view of Mitton in view of Nishimura teach all of 
the limitations as in claim 17 and 21 , above. 

Furthermore, Finn teaches further comprising storage for storing said 
media fragments (see page 1 1 , line 9, database 9 or 1 0). 

Furthermore, Mitton teaches storing the dramatic parameters ( see col. 5, 
lines 22-32, list of event and lines 35 where the MIDI file is created and is obvious 
it will be stored (see Abstract). 

As to claim 19, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 17, above. 

Furthermore, Finn teaches wherein said at least one output device 
comprises display means on which said media fragments are displayed (see 
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page 6, lines 5, monitor 4, and page 21, lines 25-26, user presented with search 
results.). 

As to claim 20, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 17, above. 

Furthermore, Finn teaches wherein said at least one output device 
comprises display means on which said media fragments are displayed (see 
page 6, lines 5, monitor 4, and page 21 , lines 25-26, user presented with search 
results.). 

Furthermore, Mitton teaches the output device responsive to instructions 
associated with said dramatic parameters (see col. 10, lines 6-9, user can modify 
the event list). 

5. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Finn in 
view of Mitton in view of Nishimura as applied to claim 1 above, and further in view of 
Weare (US 2003/0045954). 

As to claim 4, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

However, Finn in view of Mitton in view of Nishimura does not teach the 
parameters being mood, change of pace incidents. 

Weare does teach use of parameters mood (see [0095], mood), change of 
pace (see [0066], flow)) and incidents (see [0066], rhythmic activity]). 
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It would have been obvious to one skilled in the art at the time the 
invention was made to modify the audio searching of Finn in view of Mitton in 
view of Nishimura, and use video as taught by Weare, for the classification of 
media entities according to melodic properties (see Weare [0002]). 

9. Claims 6 and 11-14 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Finn in view of Mitton in view of Nishimura as applied to claim 1 above, and further 
in view of Balnaves (US 6,954,894). 

As to claim 6, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

However, Finn in view of Mitton in view of Nishimura do not specifically 
teach generating fragments. 

Balnaves teaches generating a fragment (see col. 11, lines 12-29, where 
the user input is modified to form a fragment depending on template selected, 
silent movie is chosen). 

It would have been obvious to one skilled in the art at the time the 
invention was made to modify the audio searching of Finn in view of Mitton in 
view of Nishimura, and use video as taught by Balnaves, for effectively 
controlling and editing multimedia output (see Balnaves , col. 1, lines 7-11). 

As to claim 1 1 , Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 
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Furthermore Finn in view of Mitton in view of Nishimura teach dramatic 
parameter data, matching dramatic parameters to media fragments, and 
selecting and generating according to dramatic parameter lists. 

However, Finn in view of Mitton in view of Nishimura do not specifically 
teach the story template 

Balnaves teaches prior to obtaining said media segments, selecting a 
story template (see col. 8, lines 27-30, user selects template) at least in part in 
dependence on said table of dramatic parameter (see col. 8, lines 54-60, 
templates used to evoke action or intent and see Figure 12 and 13, where each 
type of movie has a specific template), said story template affecting said 
obtaining of media fragments (see Figure 5, 501 and 508, template and movie 
player, output of processed data) (e.g. The template chosen affects the output 
data). 

It would have been obvious to one skilled in the art at the time the 
invention was made to modify the audio searching of Finn in view of Mitton in 
view of Nishimura, and use video as taught by Balnaves, for effectively 
controlling and editing multimedia output (see Balnaves , col. 1, lines 7-11). 

As to claim 12, Finn in view of Mitton in view of Nishimura in view of Balnaves 
teach all of the limitations as in claim 1, above. 
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Furthermore, Finn teaches the sue of dramatic parameters (see page 6, 
3rd and 4th paragraph, and see Figure 3A, from the cited portion and the figure it 
can be seen that a time ordered sequence of pitches are extracted) 

Furthermore, Balnaves teaches wherein said story template comprises 
dramatic parameter data related to a narrative story structure (see Figure 12 and 
Figure 13, each type of template movie selected consists of various parameters. 

As to claim 13, Finn in view of Mitton in view of Nishimura in view of Balnaves 
teach all of the limitations as in claim 1, above. 

Furthermore, Finn teaches matching the dramatic parameters with the 
media fragments features (see page 1 1 , lines 8-10, features from data are 
compared with search criteria, where the matching criteria of dramatic 
parameters is shown in page 25, lines 2 and 15), 

Furthermore, Balnaves teaches using a story template comprises dramatic 
parameter data related to a narrative story structure (see Figure 12 and Figure 
13, each type of template movie selected consists of various parameters. 

As to claim 14, Finn in view of Mitton in view of Nishimura in view of Balnaves 
teach all of the limitations as in claim 1, above. 

Furthermore, Balnaves teaches wherein the story template for selection is 
generated according to logical story structure rules and the dramatic parameter 
list (see Figures 12 and 13, where sample template is shown) (e.g. From the 
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Figure, it is obvious to one skilled in the art that the templates were generated 
beforehand. Based on movie genre or user preferences related to the genre.) 

10. Claim 15 is rejected under 35 U.S.C. 103(a) as being unpatentable over Finn in 
view of Mitton in view of Nishimura as applied to claim 1 above, and further in view of 
Williams (US 6,308,154). 

As to claim 15, Finn in view of Mitton in view of Nishimura teach all of the 
limitations as in claim 1, above. 

However, Finn in view of Mitton in view of Nishimura do not specifically 

teach the use of physical markup language tags. 

Williams teaches instruction set of a markup language (see Col. 3, lines 2- 

8, where Williams discusses attributes are encoded using a markup language 

and markup indicators). 

It would have been obvious to one skilled in the art at the time the 

invention was made to modify the audio searching as taught by Finn in view of 

Mitton in view of Nishimura, and use instruction set of a markup language as 

taught by Williams, thus allowing measurement and encoding of recognized 

content, as discussed by Williams (see Col. 1, lines 52-57). 

Conclusion 

6. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 
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Furuyama et al. (US 6,61 1 ,803) is cited to disclose retrieval of video and audio 
scene based on speech recognition. Charlesworth et al. (US 7,240,003) is cited to 
disclose annotation of video and audio database for retrieval. Li et al. (US 
2003/0107592) is cited to disclose retrieving information related to people in vide 
programs. 

Nepal et al. ("Automatic Detection of Goal segments in Basketball Videos") is 
cited to disclose detecting special segments in video for retrieval. Mihajlovic et al. 
("Automatic Annotation of Formula 1 Races for content-based Video Retrieval") is cited 
to disclose a video retrieval system using multimodal clues. Bakker et al. ("Semantic 
Video Retrieval Using Audio Analysis") is cited to disclose using audio content for 
retrieving video segments. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to PARAS SHAH whose telephone number is (571)270- 
1650. The examiner can normally be reached on MON.-THURS. 7:00a. m.-4:00p.m. 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Patrick Edouard can be reached on (571)272-7603. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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Examiner, Art Unit 2626 
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/Patrick N. Edouard/ 

Supervisory Patent Examiner, Art Unit 2626 



